User interface for multimodal information system

ABSTRACT

A method and system for providing a user interface for information services related to multimodal information on a computer system is presented. A system for providing a user interface for augmenting multimodal information with information services and interacting with the information services is described.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. provisional patentapplication 60/689,345, 60/689,613, 60/689,618, 60/689,741, and60/689,743, all filed Jun. 10, 2005, and is a continuation in part ofU.S. patent application Ser. No. 11/215,601, filed Aug. 30, 2005, whichclaims the benefit of U.S. provisional patent application 60/606,282,filed Aug. 31, 2004. These applications are incorporated by referencealong with any references cited in this application.

BACKGROUND OF THE INVENTION

The present invention relates to presenting information services on acomputer system. Specifically, it relates to the presentation ofinformation services related to multimodal information.

Systems for presenting information on a computer system commonlyreferred to as the computer user interface have been developed forvarious computer systems and computer usage scenarios. User interfacesare usually customized to the computer system under consideration andthe use for which it is intended. For instance, a desktop personalcomputer has a windowing software user interface for intuitive use ofmultiple applications simultaneously while the individual softwareapplications have their own user interfaces designed for the intendedusage of the application.

Presenting a multimodal multimedia user interface on with restrictedcapabilities presents unique challenges. An application such asaccessing information services relevant to multimodal information thathitherto has not been available, places additional requirements ondesigning a user interface for it. In view of the foregoing, a userinterface for such an application that uniquely addresses theconstraints of presenting information services related to multimodalinformation and still provides a versatile functionality is desirable.

BRIEF SUMMARY OF THE INVENTION

A user interface for a multimodal information system that providesinformation services and multimedia documents related to multimodalinformation is presented. The user interface enables users to retrieve,present, author, communicate, store and interact with the informationservices and multimedia documents provided by the multimodal informationsystem. The user interface is comprised of views designed to addressvarious functionality required for using the multimodal informationsystem. The operation of the views comprising the user interfaceincluding user inputs and the user interface responses are alsodescribed. The user interface also enables the presentation of anaugmented version of multimodal input information.

Other objects, features, and advantages of the present invention willbecome apparent upon consideration of the following detailed descriptionand the accompanying drawings, in which like reference designationsrepresent like features throughout the figures.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1( a) illustrates an exemplary embodiment of a login view.

FIG. 1( b) illustrates an exemplary embodiment of a management view.

FIG. 1( c) illustrates an exemplary embodiment of a authoring view.

FIG. 1( d) illustrates an exemplary embodiment of a capture view.

FIG. 1( e) illustrates an exemplary embodiment of an index view.

FIG. 1( f) illustrates an exemplary embodiment of a folder view.

FIG. 1( g) illustrates an exemplary embodiment of a content view.

FIG. 1( h) illustrates an alternate embodiment of a content view.

FIG. 1( i) illustrates an exemplary embodiment of a transient view.

FIG. 1( j) illustrates an exemplary embodiment of a view integrating amenu.

FIG. 1( k) illustrates an alternate exemplary embodiment of a authoringview.

FIG. 1( l) illustrates an alternate exemplary embodiment of an indexview.

FIG. 1( m) illustrates an alternate exemplary embodiment of a contentview.

FIG. 1( n) illustrates an exemplary embodiment of a statistics view.

FIG. 2 illustrates an exemplary process for navigating between views.

FIG. 3 illustrates an exemplary process for using the login view.

FIG. 4 illustrates an exemplary process for using the management view.

FIG. 5 illustrates an exemplary process for using the capture view.

FIG. 6 illustrates an exemplary process for using the index view.

FIG. 7 illustrates an exemplary process for using the folder view.

FIG. 8 illustrates an exemplary process for using the content view.

FIG. 9 illustrates an exemplary process for using the authoring view.

FIG. 10 illustrates an exemplary process for presenting informationservices in the smart mode of operation for passive augmentation.

FIG. 11 illustrates an exemplary process for presenting informationservices in the advanced options mode of operation for passiveaugmentation.

FIG. 12 illustrates an exemplary process for presenting informationservices in the user-controlled mode of operation for passiveaugmentation.

FIG. 13 illustrates an exemplary process for presenting informationservices for active augmentation.

FIG. 14 illustrates an exemplary system, in accordance with anembodiment.

FIG. 15 illustrates an alternate view of an exemplary system, inaccordance with an embodiment.

FIG. 16 is a block diagram illustrating an exemplary computer system ofsuitable for providing information services related to multimodalinformation, in accordance with an embodiment.

DETAILED DESCRIPTION OF THE INVENTION

A system and method are described for a computer informationpresentation system. The specific embodiments described in thisdescription represent exemplary instances of the present invention, andare illustrative in nature rather than restrictive.

In the following description, for purposes of explanation, numerousspecific details are set forth in order to provide a thoroughunderstanding of the invention. It will be apparent, however, to oneskilled in the art, that the invention can be practiced without thesespecific details. In other instances, structures and devices are shownin block diagram form in order to avoid obscuring the invention.

Reference in the specification to “one embodiment” or “an embodiment” or“some embodiments” means that a particular feature, structure, orcharacteristic described in connection with the embodiment is includedin at least one embodiment of the invention. The appearances of thephrase “in one embodiment” or “in some embodiments” in various places inthe specification are not necessarily all referring to the sameembodiment, nor are separate or alternative embodiments mutuallyexclusive of other embodiments. Features and aspects of variousembodiments may be integrated into other embodiments, and embodimentsillustrated in this document may be implemented without all of thefeatures or aspects illustrated or described.

An embodiment presents a computer information presentation system oruser interface that addresses the unique characteristics of presenting amultimodal multimedia user experience on computer systems, includingcomputer systems with restricted capabilities. The restrictedcapabilities of the computer systems may derive from factors such asrestricted physical size and limited input and output features. Examplesof computer systems with restricted capabilities include cellularphones, camera phones, personal digital assistants (PDAs), handheldcomputers and television remote controls. The multimedia user experiencepresented may include both information services and multimediadocuments.

In the scope of this description the term “system” is used to refer to asystem for providing information services related to multimodalinformation. The term “information service” is used to refer to a userexperience provided by the system that may include (1) the logic topresent the user experience, (2) multimedia content, and (3) relateduser interfaces. The term “content” is used to refer to multimedia dataused in the information services. The term “client device” refers to acomputer system that incorporates the user interface. The term “visualimagery” refers to multimedia content in the form of a single stillimage, a plurality of still images, a single video sequence, a pluralityof video sequences, or a combination thereof.

The user interface presented here may constitute the user interface of acomputer system in its entirety or form one component of the computersystem's user interface. In the latter case, the user interfacepresented here may form the user interface of a specific application ormodule installed in the computer system. The user interface and itslogic may be implemented in software, firmware, hardware or acombination thereof.

The user interface may be comprised of screens or panes hereafterreferred to as “views.” The views described are (1) capture (i.e.,camera) view, (2) index view, (3) folder view, (4) content view, (5)authoring view, (6) management view, (7) login view, (8) transient view,and (9) statistics view as illustrated in FIGS. 1( a)-1(n). Depending onthe implemented features some of these panes may or may not be present.Further, in some embodiments, the views may be integrated into otherfeatures available on a client device. For instance, in someembodiments, the capture view may be integrated into a camera softwareapplication available on a client device. In other embodiments, theviews may be integrated into the web browser functionality of a clientdevice. While the following primarily discusses the visual presentationof the user interface, user input in the form of textual, audio or videoinputs is accomplished through other components such as a keypad,microphone or camera integrated into client device implementing the userinterface. Further, audio information may be presented through an audiooutput device such as a speaker integrated into the client device.

User Interface Architecture

FIGS. 1( a)-1(n) illustrate the common structural attributes ofexemplary views that may be integrated into embodiments. In someembodiments, where a plurality of views is available for presentation inthe user interface, the availability of the plurality of views may bepresented in the form of tabbed panel 122 (shown with dashed ellipse).

Tabbed panel 122 includes a tab representing each view, as illustratedin FIGS. 1( b) through 1(j). The tab for the view that is currentlyselected and active 134 may be presented with a distinct representationto distinguish it from the other nonactive tabs. Tabs may be markedeither with graphics and/or text to indicate the identity of each view.This tabbed layout for the views serves as a metaphorical representationof the views being arranged in an overlapping stack with the active viewpresented prominently on the display.

The nonactive views may either be completely hidden or presented in aminimized representation. Some embodiments may also include userinterface elements on the views to represent various activity states inthe system such as activity indicator 124 and progress indicator 126.

FIG. 1( a) illustrates an exemplary embodiment for a “login” view 110 ofthe user interface. The login view enables users to enter authenticationparameters such as a username and password for authentication into asystem implementing the user interface. The login view may include textfields for entering an alphanumeric user identifier 112 and password114. The authentication process may either be initiated by clicking on alogin button 116 or by choosing a similar option for the menu systemintegrated into the view. In one embodiment, the login operation isautomatically launched without any user input using authenticationparameters stored in the system.

FIG. 1( b) illustrates an exemplary embodiment for a “management” or“settings” view 120 of the user interface. The management view presentsa mechanism for managing the features of the system such as an addressbook, lists of friends of user, user groups and their access privileges,account management features such as user names, passwords andsubscription levels and user preferences such as user interface,authoring, information classification and presentation preferences. Thevarious features and available options for the features may be presentedin a list representation. User may edit or change the settings for thefeatures using menu options or alphanumeric inputs.

FIG. 1( c) illustrates an exemplary embodiment for an “authoring” view130 of the user interface. The authoring view enables the user to authornew content and information services. The newly authored content mayinclude content in audio, visual, textual and graphical media formats.To enable the authoring of content and information services, theauthoring view incorporates authoring tools such as a text editor fortextual content 132, visual capture controls such as record, pause, stopand edit for authoring still picture and moving video information, audiocapture controls such as record, pause, stop and edit for authoringaudio information and drawing tools for authoring graphical information136. In addition, the authoring view incorporates appropriate controlsfor presenting, inputting and editing metadata such as the time,location and access privileges for the authored information 138.

FIG. 1( d) illustrates an exemplary embodiment for a “capture” view 140of the user interface. The capture view presents a means of displayinglive visual imagery and/or recorded visual imagery. When presenting livevisual imagery, the capture view displays live video imagery capturedfrom a visual sensor integrated into the device such as a camera. Whilepresenting recorded visual imagery, the capture view displaysprerecorded video imagery, either in the form of still images or motionvideo from a storage medium. The capture of the live visual imagery mayalso be aided through the use of cues for maintaining the alignment andsize of the captured visual imagery as illustrated by the dottedreference lines 142 overlaid on the visual imagery which is presented asa viewfinder 144.

The visual imagery may be a single still image, a sequence of stillimages or a video sequence. The capture view optionally integratescontrols such as play, stop, pause, and edit for controlling thepresentation of the recorded visual imagery 148. Also additionalcontrols for operations such as start, stop, record, optical zoom,digital zoom, manual focus, automatic focus, white balance andspecialized filters for manipulating the capture and presentation oflive visual imagery may be present (not shown). These controls rely onthe capabilities of the underlying client device to translate the userinputs at the user interface into appropriate commands for the clientdevice.

In some embodiments, the functionality of the capture view may beprovided by a camera software application integrated into the clientdevice. In some embodiments, the capture view may also include controls(not shown) for capturing other multimodal inputs such as audio andtextual inputs. Examples include record/stop controls for controllingthe capture of the audio input and a text entry box for entering textualinformation using a keypad integrated into the client device. In someembodiments, the capture view provides feedback on the quality (i.e.,lighting, camera jitter, ambient noise level, etc.) of the capturedinputs using appropriate controls (not shown).

In addition, the capture view may incorporate a mechanism for augmentingthe visual imagery presented with audio, visual, textual, and graphicalinformation along with a mechanism for controlling the presentation ofsuch information. This is represented for instance by the dashedrectangular box graphic 146 overlaid on the visual imagery 144. Thevisual, textual and graphical augmentation information may be presentedas a standalone entity on the user interface, overlaid on top of thevisual imagery or embedded into the visual imagery such as to become anindistinguishable part of it. In addition, audio augmentationinformation may be presented through speakers and tactile augmentationinformation through tactile transducers respectively.

Besides presenting such augmentation information, the capture view mayalso include controls (not shown) for authoring or editing suchaugmentation information either directly in the capture view or in theauthoring view. This enables the user to annotate or augment the visualimagery for further use and manipulation. The authoring and editingcontrols that can optionally be integrated into the capture view includea means of textual information entry such as text box widget to captureadditional textual inputs from the user, a free form graphical drawing“pen” and an interface to record audio information through a microphoneintegrated into a client device. The capture view may also providecontrols (not shown) for highlighting or marking up portions of thevisual imagery. For instance, text in the visual imagery displayed onthe viewfinder 144 may be highlighted for obtaining additionalinformation about the text.

FIG. 1( e) illustrates an exemplary embodiment for an “index” view 150of the user interface. The index view 150 presents one or more availableinformation service options. The individual entries in the list ofinformation service options may incorporate information in audio,visual, textual and graphical media formats. Optionally, the entries inthe list may also include indicators representing associated metadatasuch as the time, location, author 152, information media formats 156and access privileges 158.

The individual entries in the list may be formatted to present theinformation services optimally in the available space on the display.Typically, the individual entries include a short text string which maybe the title of a web page or a snippet of the textual information thatis presented in its entirety in content view 170. The individual entriesmay also include a “thumbnail” view of the content and informationservices presented in its entirety in the content view.

Optionally, the index view 150 may incorporate a preview pane 154 thatdisplays additional information relevant to the entry currently selectedin the list. The additional information displayed in the preview panemay be textual or graphical and may include details such as an excerptfrom the content and information services to be displayed in the contentview 170, an abstract from the content and information services to bedisplayed in the content view 170 or metadata associated with theselected entry. The preview pane may be displayed as an overlappingregion on top of the list or as a nonoverlapping region adjacent to thelist. The index view 150 optionally also incorporates controls forcontrolling the presentation of the information services such as play,pause and stop controls for audio, video and animated graphicalinformation (not shown).

In addition, the index view may incorporate controls (not shown) forcommunicating and storing the information services list presented in theview or a selected information service in the list. The informationservices list may be communicated using e-mail, SMS, MMS, fax, datanetworks, or circuit-switched networks. The information services listmay also be stored on a system server or desktop personal computer forlater access through the client user interface or through other meanssuch as a web browser on a personal computer. Controls for control,communication and storage of information services (not shown) may beintegrated with a menu system or integrated into index view 150.

Further, in some embodiments, index view 150 may incorporate controlsfor refining the list of information services provided (not shown). Suchcontrols may be in the form of a text entry field where the user caninput additional textual information for refining the list or thecontrols may be in the form of audio information capture through amicrophone integrated in to the client device for input of therefinement inputs in audio format.

The system may use speech recognition to convert refinement inputs inaudio format to a machine readable form. The refinement inputs may beused as an input to refine or filter the entries presented in the indexview.

Refinements may be based on criteria such as the type of an informationservice (regular, commercial, sponsored, etc.), the content of aninformation service, metadata of an information service or others. Inembodiments that support textual input of the refinement information,the text input control may be presented adjacent to the list or in amanner overlapping the list completely or partially. The text inputcontrol may also contain predefined textual information generated by theclient.

For instance, parameters used to generate the list of informationservices presented in index view 150 may be presented in the text inputcontrol. Further, the list presented in index view 150 may have one ormore columns. In some embodiments, the list may not be structured into aregular geometric layout and may have arbitrary positioning on thedisplay.

FIG. 1( f) illustrates an exemplary embodiment for a “folder” view 160of the user interface. The folder view presents a visual mechanism fororganizing the information services displayed in index view 150. Theinformation services can be organized into hierarchical collectionscalled folders. The folders are referred to by an associated title. Thefolder view 160 presents a visual representation of the collection offolders at a chosen level in the hierarchy.

The organization of the information services into folders is performedeither manually by users of the interface or automatically by the systemusing parameters such as the analysis of the content and informationservices, user preferences or past user behavior in using the system. Insome embodiments such folders may be created by users using textual“tags.” The folders may also be communicated using e-mail, SMS, MMS,fax, data networks, or circuit-switched networks. In some embodiments,folder view 160 may include controls for refining the list of folderspresented using inputs such as text or audio.

FIG. 1( g) illustrates an exemplary embodiment for a “content” view 170of the user interface. The content view 170 presents informationservices in their entirety for consumption by the user. The content view170 includes support for the presentation of information in audio,visual, textual, and graphical formats 172 as well as controls 174 suchas play, pause, and stop controls for controlling the presentation ofinformation services that are presented over a time duration as in thecase of audio, video, or animated graphical information. In someembodiments, content view 170 may include controls for refining theinformation services presented.

FIG. 1( h) illustrates an alternate exemplary embodiment for a “content”view 180 of the user interface. The content view 180 presentsinformation services such as to maximize the use of the availabledisplay space for presenting information services. In contrast tocontent view 170, content view 180 has fewer controls displayed on thedisplay to maximize the display space available for presentinginformation services.

Information services presented in content view 170 and content view 180may be formatted for a concise and intuitive presentation of theinformation. Thus, extended textual content, such as the content from aweb page, which may be cumbersome to read on a small form factor screen,is abridged in the form of excerpts and summaries for presentation.

In addition, a “thumbnail” picture representation of the information asit would be presented on a full featured visual display environment suchas a personal computer based web browser might be included in thecontent view to provide a hint to the original formatting of theinformation prior to its reformatting. When such a thumbnail picture isincluded, portions of the thumbnail picture that represent theinformation presented elsewhere in the content view might optionally behighlighted. Similarly, other content and information services may bereformatted to suit the presentation capabilities of a computer systemimplementing the user interface.

In some embodiments, hyperlinked content and information services may bepresented in the content view. Hyperlinked content is presented in thecontent view, similar to its representation on a full featuredenvironment such as a personal computer based web browser. In this modeof presentation, the presented content and information services includeembedded hyperlinks that may be traversed by selecting and activatingthem. Activating the hyperlinks may result in the presentation of thecontent and information services hyperlinked to the content orinformation service presented currently. The selection and activation ofthe hyperlinks may be performed using various input componentsintegrated into the client device. Activating a hyperlink may result inthe presentation of the hyperlinked content using other components ofthe client device such as a web browser.

Hyperlinked content and information services may also be reformatted andpresented in the content view such that hyperlink traversal ispotentially reduced or minimized. For instance, a plurality of contentand information services linked to a specific content or informationservice through hyperlinks may be reformatted into nonhyperlinked formatsuch that the top level content and information service is presentedearlier followed sequentially by the linked content and informationservices, rather than requiring a user to follow the links individually.

For instance, in the case of a textual content hyperlinked to aplurality of other textual content, a single “flat” textualrepresentation may be presented where the top level textual content isfollowed by the content from the other textual content hyperlinked withthe top level one. Such reformatting of hyperlinked content andinformation services may also be combined with other reformattingtechniques such as the abstract and excerpt generation describedearlier. The reformatted hyperlinked content and information servicesmay play a role in the presentation of content sourced from the WorldWide Web and the sponsored information services.

The content view 170 may also provide controls (not shown) forhighlighting or marking up portions of the displayed information, forinstance the highlighting of text or an image, for obtaining additionalinformation about the highlighted information or for augmenting thehighlighted information with other user authored or system generatedinformation. The content view 170 may also include controls to demarcatesections of multimedia content and to select regions of the demarcatedmultimedia content in order to obtain additional information or toaugment it with user authored information.

Alternately, the multimedia content may incorporate demarcationsinnately and content view controls may enable selection of suchpredemarcated content sections. For example, it might be possible toselect a human form in a video delineated by an outline by clicking on ajoystick or button. Similarly it may be possible to select a segment ofaudio by clicking on the joystick to mark the start and end points ofthe audio segment.

In addition, the content view 170 may also incorporate controls (notshown) for communicating and storing the information presented in theview. The information service presented in the content view may becommunicated using e-mail, SMS, MMS, fax, data networks, orcircuit-switched networks or stored on a central server or desktoppersonal computer for later access through the user interface presentedhere or through other means such as a web browser.

FIG. 1( i) illustrates an exemplary embodiment for an “interim content”or “transient” view 190 of the user interface. The transient view 190may be presented on the client device display when the system istransitioning between different views or when the system is busy with anoperation over an extended duration of time. Thus, in some embodiments,the transient view presents interstitial information services, i.e.,information services that are provided in between other system activity.For instance, transient view 190 may be presented when the system isanalyzing multimodal inputs for identifying relevant informationservices. Transient view 190 may be used to present information servicesincluding content in text, audio, video and other media types.

FIG. 1( j) illustrates an exemplary embodiment 200 for menus used in theuser interface. The user may also initiate commands for the system usingmenu 202. In some embodiments, menu 202 may be presented as an overlayon top of the views. In some embodiments, menu 202 may be presentedadjacent to other elements of the user interface. In some embodiments,the menus may not have a visual representation and may be controlledthrough audio inputs.

FIG. 1( k) illustrates an alternate exemplary embodiment of an authoringview 210 for use on a full-featured computer system such as a personalcomputer. For example, in some embodiments, information services may beauthored using various content and information service logic providedthrough alternate authoring view 210 using a web browser integrated intoa personal computer. Optionally, alternate authoring view 210 may alsoallow uploading of information services including associated logic andcontent. In some embodiments, the functionality of alternate authoringview 210 may be integrated into a web browser on a personal computer,for instance using a toolbar.

FIG. 1( l) illustrates an alternate exemplary embodiment of an indexview 220 for use on a full-featured computer system such as a personalcomputer. For example, a web browser on a personal computer may be usedto present the information in index view 150 with greater details. Suchan alternate index view may optionally present a means to access ahistorical record of all information presented through the system. Inanother embodiment, the alternate index view information may bepresented in the form of an e-mail message on an e-mail application.

FIG. 1( m) illustrates an alternate exemplary embodiment of a contentview 230 for use on a full-featured computer system such as a personalcomputer. For example, a web browser on a personal computer may be usedto present the information in content view 170 or 180 with greaterdetails. Such an alternate content view may optionally present a meansto access a historical record of all information presented through thesystem. In another embodiment, the alternate content view informationmay be presented in the form of an e-mail message on an e-mailapplication.

FIG. 1( n) illustrates an exemplary embodiment of a statistics view 240.This view presents historical data on usage of the service. Users mayuse this view to view their use of the system over time. For instance,they can view all past queries for content made through the system orall past multimodal inputs they stored in the system.

Operation

The views of the user interface discussed above are presented on adisplay integrated into a client device. In some embodiments, the userinterface may not be active at all times. In such embodiments, the userinterface may be activated through user input or by the system based onsystem-determined criteria.

An exemplary embodiment of such a client device is a cellular phone,which incorporate a number of software applications in an operatingsystem environment. In some embodiments, a user may activate theapplication incorporating the user interface by pressing a dedicatedbutton on the client device, pressing and holding a dedicated button onthe device or selecting and activating an icon in a graphical userinterface.

In some embodiments, the system may automatically activate thefunctionality of the system incorporating the user interface bymonitoring system parameters such as client device motion, changes inlighting, time of day, etc.

In some embodiments, the application or functionality incorporating theuser interface is activated through a communication message such as anSMS received by the client device. Upon activation, one of the views ofthe user interface is presented and user interaction is continued. Insome embodiments, the communication message may also be generated by thesystem upon receiving responses to other inputs initiated by the user,such as a SMS message.

A user may interact and control the features of the user interface usinginput components integrated with the client device such as keypad,joystick, thumb wheel and other pointing and cursor control inputmechanisms. A user may have the system present the various viewsdescribed earlier using appropriate inputs. For instance, on a deviceequipped with a joystick input, the user may switch between the adjacentviews by flicking or moving the joystick to the left or right to selectand display the view in the corresponding direction.

Upon such selection of a view, the active view is displayed on the userinterface and tabbed panel 122 updated to reflect the view that iscurrently active. In some embodiments, the system may present viewsautomatically based on system determined criteria. For instance,transient view 190 may be presented by the system automatically when asystem operation is being performed over an extended duration of time.The user may also initiate commands for the system using menus 192. Menu192 may be presented when triggered through a user input component suchas keypad or soft key entry or automatically by the system underpredefined system states.

When the system is processing or communicating information, activityindicator 124 maybe activated. In some embodiments, activity indicator124 is represented in the form of an icon that flashes to indicate thebusy state of the system. In some embodiments, the progress of theexecution of an extended operation may be represented by a progressindicator 126 that is updated to reflect the extent of completion of theoperation. The progress indicator may be in the form of a linear striprepresenting the extent of the operation in its entirety, overlaid witha second linear strip of a different color that denotes the extent ofthe completion of the operation.

The dark portion of the progress indicator 126 representing the extentof the operation in its entirety which is overlaid by the light portionof progress indicator 126 representing the extent of completion of theoperation. In some embodiments, activity indicator 124 and progressindicator 126 may be represented by other graphical representations. Insome embodiments, activity indicator 124 and progress indicator 126 maybe presented using output components other than the client devicedisplay.

Capture view 140 enables the user to capture multimodal information toinitiate interaction with the system. Information services identifiedand presented as relevant to multimodal information by the system may bepresented independent of the multimodal information or such as toaugment the multimodal information. In embodiments where informationservices are presented independent of the multimodal information, theinformation services may be presented using index view 150, folder view160, and content view 170 and 180. In embodiments where informationservices are presented such as to augment the multimodal information,the presentation of the information services can be in one of two modes:(1) passive augmentation mode and (2) active augmentation mode.

In the passive augmentation mode, the user captures multimodalinformation using the capture view. The user may use the multimodalinformation to access relevant information services using the followingprocesses: (1) request the system to automatically present an augmentedversion of the multimodal information (smart mode) or (2) present theuser with a menu of augmentation options integrated with the multimodalinformation from which the user can select (advanced options mode) or(3) markup areas of the multimodal information for which augmentationinformation services can then be requested (user-controlled mode).

In some embodiments operating in the smart mode of operation, theaugmentation information services are automatically generated andpresented by the system upon capture of the multimodal informationwithout additional user input. In some embodiments operating in thesmart mode of operation, the augmentation information services aregenerated and presented by the system upon explicit request by the userthrough inputs such as a key press or joystick click following thecapture of the multimodal information.

In the advanced options mode of operation, the augmentation options maybe presented in the capture view in the form of graphical, textual, oraudio overlays and the user may then select among the availableaugmentation options. The selection of an augmentation option maytrigger the presentation of an extended list of information services inindex view 150, the presentation of a information service in contentview 170 or 180 or the authoring and association of new content andinformation services to augment the multimodal information in authoringview 130.

In some embodiments operating in the user-controlled mode of operation,the user explicitly marks up the region of the captured multimodalinformation for which he seeks augmentation information and thenrequests relevant augmentation information services with an explicitinput such as a key press or joystick click.

The 3 modes of passive augmentation may be illustrated by the followingexample scenario. A still image of a page of text is captured anddisplayed in capture view 140. In the smart mode of operation,information services such as links to information on the World Wide Webrelevant to the captured textual information and dictionary lookup forwords in the textual information are automatically presented to the useror upon request by the user in the index view. The user request may beinput using a key on the client device.

In the advanced options mode of operation, the user requests theaugmentation information options upon which augmentation information,for instance, in the form of graphical highlights for the words in thetextual information is overlaid on top of the image. The user may thenselect a highlighted word or a set of highlighted words for which hewants to request information services and request associated informationservices.

In the user-controlled mode of operation, the user selects a region ofthe visual imagery using a drawing cursor integrated in to the userinterface and then requests associated information services. Audioaugmentation information may be presented through an audio output devicesuch as a speaker integrated into the client device.

In some embodiments operating in the active augmentation mode ofoperation, the augmentation of the visual imagery happens without anyexplicit user input. When the user points the camera built into a clientdevice at a scene, the live visual imagery sensed by the camera isdisplayed in the capture view viewfinder 144. Simultaneously,augmentation content and information services are overlaid on the visualimagery without any additional user input to request the augmentationinformation services.

For instance, if the camera is scanned over a page of text, some wordsmay appear highlighted, changed in color or be augmented in other formsin viewfinder 144 to indicate the availability of information servicesrelevant to the words. In some embodiments operating in the activeaugmentation mode of operation, as the camera is scanned over a scene,the system may recognize the availability of augmentation informationservices for certain regions of the scene and alert the user with anaudio beep along with highlighting of the regions.

In some embodiments operating in the active augmentation mode ofoperation, the augmentation may be presented as icons, emoticons orother graphical symbols and marks. Thus, in the active augmentation modeof operation, augmentation information services are presented as thevisual imagery is being captured in contrast to the passive augmentationmode of operation, where augmentation information services are presentedafter the visual imagery is captured. The active augmentation mode ofoperation is also applicable to visual imagery obtained from storage.

In some embodiments, interim content and information services may bepresented on transient view 190 while the system is busy with anoperation over an extended duration of time or on other instances asdetermined by the system. The information services presented intransient view 190 may include sponsored information services orinformation on the state of the system. The information servicespresented in transient view 190 may or may not be relevant to themultimodal input information and other information services beingaccessed by the user through the system. Examples of interim contentinclude simple information such as news articles, advertisements,quotations, tip of the day, or other more advanced information services.

Some embodiments of the user interface may be understood with referenceto the process of a user's interaction with the system through the userinterface. FIG. 2 illustrates an embodiment of a process of operatingthe user interface. Process 1200 includes presenting a view of the userinterface, receiving a view change signal and changing the viewpresented on the user interface, receiving a selection of a part of theuser interface and displaying that selection, and receiving a featureselection and operating the selected feature.

Process 1200 and other processes of this description are implemented asa set of modules, which may be process modules or operations, softwaremodules with associated functions or effects, hardware modules designedto fulfill the process operations, or some combination of the varioustypes of modules. The modules of process 1200 and other processesdescribed herein may be rearranged, such as in a parallel or serialfashion, and may be reordered, combined, or subdivided in variousembodiments.

Process 1200 initiates with display of the user interface at module1210. Thus, the active view of the user interface may be presented inthe foreground of the user interface. The tab corresponding to theactive view in tabbed panel 122 may be presented with a representationdenoting its active state.

At module 1220, a request to change the view presented on the userinterface is received. This may be a user input such as a key press orjoystick input, as described above. At module 1230, the view presentedin the user interface is changed, such as to the next view. A user maycycle through the various views of the user interface due to repetitionof these two modules (1220 and 1230).

At module 1240, features of the user interface are selected. Inparticular, a single feature or option is selected at any given time.The corresponding functionality for the feature is then operated orexecuted at module 1250. This may result in various actions, such asaugmentation of multimodal information, following a link, accessingdetailed views of information services, or capturing multimodalinformation, for example.

FIG. 3 illustrates an embodiment of a process 1300 of authenticating tothe system using login view 110 integrated into a client. Uponpresentation of login view on the display 1310, the user enters atextual user identifier and password 1320 into the respective textfields 112 and 114. The user then highlights the login button 116 andactivates by clicking on a joystick. The client then encodes andcommunicates the authentication information to the system server 1330.

FIG. 4 illustrates an embodiment of a process 1400 of managing userpreferences using management view 120 integrated into a client. Uponpresentation of management view 120 on the display 1410, the userselects various options for user preferences for modification 1420.Optionally, this process may include the management of a list of friendsof the user. The user may finalize the changes by activating anappropriate menu entry. The client then encodes and communicates themodification information to the system server as required 1430.

FIG. 5 illustrates an embodiment of a process 1500 of using the captureview 140 integrated into a client. Upon presentation of capture view 140on the display 1510, a live rendering of the visual imagery captured bythe camera is displayed as a viewfinder 144 with reference marks 142 foraiding the users in capturing visual imagery with the correct resolutionand orientation.

A user may align the visual imagery in the viewfinder to the referencemarks through rotation and motion of the camera relative to the scenebeing imaged 1520. The user may also use graphical indicators for thecharacteristics of the captured visual imagery 148 to ensure the visualimagery is captured with appropriate quality required by the system1530. The user may then initiate the communication and processing of thecaptured visual imagery using appropriate inputs on the client devicekeypad or joystick 1540.

Relevant information services may optionally be then presented as anaugmentation on the visual imagery 1550. The graphical marks 146 thatare part of the information services augmenting the visual imagery maybe in the form of rectangles surrounding the regions of interest, achange in the hue, saturation or brightness of the area in and aroundthe regions of interest, change in the font and emphasis of textualelements, icons placed near the regions of interest, or other suchmarks. In some embodiments, information services may be presented 1560in index view 150 or content view 170 and 180.

FIG. 6 illustrates an embodiment of a process 1600 of using the indexview 150 integrated into a client. Upon presentation of index view 140on the display 1610, a list of information services relevant to aselected context may be presented. The user may use the cursor controlor pointing input components integrated into the client device to selectone of the information service options 1620.

Upon activation of the selected information service 1630, theinformation service may be presented 1640 in content view 170 and 180 orin the authoring view 130. Users may optionally input refinementinformation in textual or audio format and have the list of informationservices presented updated automatically. This enables incrementalrefinement of the information services presented using the additionaluser input. The refinement process may also require the use ofadditional menu options or other controls such as a button press.

FIG. 7 illustrates an embodiment of a process 1700 of using the folderview 160 integrated into a client. Upon presentation of folder view 150on the display 1710, a list of folders used to classify informationservices may be presented. The user may use the cursor control orpointing input components integrated into the client device to selectone of the folders 1720.

Upon activation of the selected folder, information services in thefolder may be presented 1730 in index view 150, content view 170 and 180or in the authoring view 130. If a folder contains nested folders thenthe contained folders are presented 1740 in folder view 160. Optionally,a user may also use the folder view to classify information services ormanage the classifications.

FIG. 8 illustrates an embodiment of a process 1800 of using the contentview 170 integrated into a client. Upon presentation of content view 170on the display 1810, an information service selected in the capture orindex views, is presented. The user can use the cursor control orpointing input components integrated into the client device for extendedinteraction with the features of the information service 1820. Forinstance, the presented information may be scrolled using cursor controlor pointing input components and multimedia content included in theinformation services may be controlled using the available controls.

FIG. 9 illustrates an embodiment of a process 1900 of using theauthoring view 180 integrated into a client. Upon presentation ofauthoring view 170 on the display 1910, a user may author a newinformation service using the presented controls 1920. The user may useone or more entities of audio, visual, textual or graphical informationto compose the new information service.

For instance, after activating the audio record control he may inputaudio information through a microphone integrated into the clientdevice. Textual inputs can be input through a keypad or keyboardintegrated into the client device. After entering the inputs, the usermay then activate menu commands 1930 to compose a new informationservice from the inputs and associate it with a context selected incapture view 140.

FIG. 10 illustrates an embodiment of a process 2000 for presentinginformation services in the smart mode of operation of passiveaugmentation of visual imagery. A user captures multimodal inputs 2010and the captured visual imagery is presented on the capture view of theuser interface 2020. The user then requests related information services2030 and is presented the augmentation information services along withthe visual imagery 2040.

FIG. 11 illustrates an embodiment of a process 2100 for presentinginformation services in the advanced options mode of operation ofpassive augmentation of visual imagery. A user captures multimodalinputs 2110 and the captured visual imagery is presented on the captureview of the user interface 2120. The user then requests relatedaugmentation information options 2130 and is presented the augmentationinformation service options 2140. The user then selects one or moreaugmentation information options 2150 and requests related informationservices 2160. Then, the user is presented with the related informationservices 2170.

FIG. 12 illustrates an embodiment of a process 2200 for presentinginformation services in the user-controlled mode of operation of passiveaugmentation of visual imagery. A user captures multimodal inputs 2210and the captured visual imagery is presented on the capture view of theuser interface 2220. The user then marks up regions of interest in thevisual imagery 2230 and requests related augmentation informationservices 2240. Then the user is presented the related augmentationinformation services 2250.

FIG. 13 illustrates an embodiment of a process 2300 for presentinginformation services with active augmentation of visual imagery. A usercaptures multimodal inputs 2310 and the captured visual imagery ispresented on the capture view of the user interface 2320. Along with thecaptured visual imagery the system automatically retrieves relatedaugmentation information services and presents in the client userinterface 2330.

The user interface described may be used with a variety of computersystems for purposes of accessing information services. Informationservices related to multimodal information provided by the system mayinclude information and optionally features and instructions for thehandling of information. As used herein, the term “informationassociated with an information service” may refer to the informationincluded in an information service.

Information services may enable the delivery, creation, deletion,modification, classification, storing, sharing, communication, andinterassociation of information. Further, information services may alsoenable the delivery, creation, deletion, modification, classification,storing, sharing, communication, and interassociation of otherinformation services. Furthermore, information services may also enablethe control of other physical and information systems in physical orcomputer environments.

As used herein, the term “physical systems” may refer to objects,systems, and mechanisms that may have a material or tangible physicalform. Examples of physical systems include a television, a robot or agarage door opener. As used herein, the term “information systems” mayrefer to processes, systems, and mechanisms that process information.Examples of information systems include a software algorithm or aknowledge base. Furthermore, information services may enable theexecution of financial transactions. Information services may containone or more data/media types such as text, audio, still images andvideo.

Further, information services may include instructions for one or moreprocesses, such as delivery of information, management of information,sharing of information, communication of information, acquisition ofuser and sensor inputs, processing of user and sensor inputs and controlof other physical and information systems.

Furthermore, information services may include instructions for one ormore processes, such as delivery of information services, management ofinformation services, sharing of information services and communicationof information services. Information services may be provided fromsources internal to the system or external to the system.

Sources external to the system may include the Internet. Examples ofInternet services include World Wide Web, e-mail and the like. Anexemplary information service may comprise of a World Wide Web page thatincludes both information and instructions for presenting theinformation. Examples of more complex information services include Websearch, e-commerce, comparison shopping, streaming video, computergames, podcasts, and the like. In another example, an informationservice may provide a modified version of the information or contentfrom a World Wide Web resource or URL.

In some embodiments, delivery of information services may includeproviding the spatial and temporal formatting and layout information forthe information services. Similarly, in some other embodiments, deliveryof information associated with information services may includeproviding the spatial and temporal formatting and layout information forthe information associated with information services.

In some embodiments, information services may include controls forgenerating various commands and activating functionality provided by thesystem. In some embodiments, information services may be provided inconjunction with visual imagery in the form of overlays or embeddedinformation services for an “augmented reality” experience.

In other embodiments, information services may be presented independentof the visual imagery. In some embodiments, information services areprovided upon request. In other embodiments, information services areprovided upon the occurrence of predefined events or upon the meeting ofpredefined criteria.

In some embodiments, information services include features that enablethe creation, deletion, modification, classification, storage andsharing of information and other information services. In someembodiments, access to information, information services and theirclassifications may be restricted to select users using theauthentication, authorization, and accounting (AAA), user groups, anddigital rights management (DRM) features included in informationservices.

In some embodiments, the classifications of information services andinformation associated with information services may be managed using afolder hierarchy. In some embodiments, information and informationservices may be communicated to recipients (e.g., other users of system2400 and other third party entities external to system 2400) throughcommunication mechanisms (e.g., SMS, e-mail, instant messaging, voicecalls, video calls, and the like). A voice call initiated with visualimagery as input is an example of an information service incorporatingfeatures for communicating information. In some embodiments,interassociations may be established between information servicesthrough hyperlinks embedded in information services.

In other embodiments, interassociations may be established betweeninformation associated with information services using hyperlinksembedded in information services. Information services may be used byusers or other physical and information systems. For example, aninformation service may switch a television to a specific channel. Insome embodiments, instructions included in information services mayactivate various user interface controls and functionality integratedinto the client.

In other embodiments, instructions included in information services mayadd new controls and functionality to the client or modify existingcontrols and functionality on the client. In some other embodiments,information services may also be synthesized from a plurality of otherinformation services.

Information services are associated with multimodal information throughinterpretation of context constituents associated with the multimodalinformation. Context constituents associated with multimodal informationmay include: 1) embedded elements derived from the multimodalinformation, 2) metadata and user inputs associated with the multimodalinformation, and 3) relevant knowledge derived from knowledge bases.

System Architecture

FIG. 14 illustrates an exemplary system, in accordance with anembodiment. Here, system 2400 includes client device 2402, communicationnetwork 2004, and system server 2406.

FIG. 15 illustrates an alternative view of an exemplary system, inaccordance with an embodiment. System 2400 illustrates the hardwarecomponents of the exemplary embodiment (e.g., client device 2402,communication network 2404, and system server 2406). Here, client device2402 communicates with system server 2406 over communication network2404. In some embodiments, client device 2402 may include camera 2502,microphone 2504, keypad 2506, touch sensor 2508, global positioningsystem (GPS) module 2510, accelerometer 2512, clock 2514, display 2516,visual indicators (e.g., LEDs) and/or a projective display (e.g., laserprojection display systems) 2518, speaker 2520, vibrator 2525, actuators2524, IR LED 2526, radio frequency (RF) module (i.e., for RF sensing andtransmission) 2528, microprocessor 2530, memory 2532, storage 2534, andcommunication interface 2536.

System server 2406 may include communication interface 2538, machines2540-2550, and load balancing subsystem 2552. Data flows 2554-2556 aretransferred between client device 2402 and system server 2406 throughcommunication network 2404.

Examples of client device 2402 may include communication equipment(e.g., cellular telephones), business productivity gadgets (e.g.,personal digital assistant (PDA)), consumer electronics devices (e.g.,digital camera and portable game devices or television remote control)and personal computers. In some embodiments, components, features, andfunctionality of client device 2402 may be integrated into a singlephysical object or device such as a camera phone.

In some embodiments, client device 2402 is a single physical device(e.g., a wireless camera phone). In other embodiments, client device2402 may be implemented in a distributed configuration across multiplephysical devices. In such embodiments, the components of client device2402 described above may be integrated with other physical devices thatare not part of client device 2402.

Examples of physical devices into which components of client device 2402may be integrated include cellular phone, digital camera, Point-of-Sale(POS) terminal, webcam, personal computer keyboard, television set,computer monitor, and the like. Components (i.e., physical, logical, andvirtual components and processes) of client device 2402 distributedacross multiple physical devices are configured to use wired or wirelesscommunication connections among them to work in a unified manner. Insome embodiments, client device 2402 may be implemented with a personalmobile gateway for connection to a wireless Wide Area Network (WAN), adigital camera for capturing visual imagery and a cellular phone forcontrol and display of information services with these componentscommunicating with each other over a wireless personal area network suchas Bluetooth™ or a local area network technology such as Wi-Fi (i.e.,IEEE 802.11x). In some other embodiments, components of client device2402 are integrated into a television remote control or cellular phonewhile a television is used as the visual output device.

In still other embodiments, a collection of wearable computingcomponents, sensors and output devices (e.g., display equipped eyeglasses, virtual retina displays, sensor equipped gloves, and the like)communicating with each other and to a long distance radio communicationtransceiver over a wireless communication network constitutes clientdevice 2402. In other embodiments, projective display 2518 projects thevisual information to be presented on to the environment and surroundingobjects using light sources (e.g., lasers), instead of displaying it ondisplay panel 2516 integrated into the client device.

While the visual components of the user interface are presented throughdisplay 2516, audio components of the user interface may be presentedthrough speaker 2520 integrated into client device 2402 while theintegrated camera 2502, microphone 2504 and keypad 2506 act as the inputsources for visual, audio and textual information. The client logic byitself may be implemented as software executing on microprocessor 2530or using equivalent firmware or hardware.

Communication network 2404 may be implemented using a wired networktechnology such as Ethernet, cable television network (DOCSIS), phonenetwork (xDSL) or fiber optic cables. Communication network 2404 mayalso use wireless network technologies such as cable replacementtechnologies such as wireless IEEE 1394, personal area networktechnologies such as Bluetooth™ Local Area Network (LAN) technologiessuch as IEEE 802.11x, wide area network (WAN) technologies such as GSM,GPRS, EDGE, UMTS, CDMA One, CDMA 1x, CDMA 1x EV-DO, CDMA 1x EV-DV, IEEE802.x networks, or their evolutions.

Communication network 2404 may also be implemented as an aggregation ofone or more wired or wireless network technologies. In some embodiments,communication network 2404 may be realized as a computer bus (e.g., PCI)or cable connection (e.g., Firewire).

FIG. 16 is a block diagram illustrating an exemplary computer systemsuitable for providing information services relevant to multimodalinformation. In some embodiments, computer system 2600 may be used toimplement computer programs, applications, methods, or other software toperform the above-described techniques for providing informationservices relevant to multimodal information such as those describedabove.

Computer system 2600 includes a bus 2602 or other communicationmechanism for communicating information, which interconnects subsystemsand devices, such as processor 2604, system memory 2606 (e.g., RAM),storage device 2608 (e.g., ROM), disk drive 2610 (e.g., magnetic oroptical), communication interface 2612 (e.g., modem or Ethernet card),display 2614 (e.g., CRT or LCD), input device 2616 (e.g., keyboard), andcursor control 2618 (e.g., mouse or trackball).

According to some embodiments, computer system 2600 performs specificoperations by processor 2604 executing one or more sequences of one ormore instructions stored in system memory 2606. Such instructions may beread into system memory 2606 from another computer readable medium, suchas static storage device 2608 or disk drive 2610. In some embodiments,hard-wired circuitry may be used in place of or in combination withsoftware instructions to implement the system.

The term “computer readable medium” refers to any medium thatparticipates in providing instructions to processor 2604 for execution.Such a medium may take many forms, including but not limited to,nonvolatile media, volatile media, and transmission media. Nonvolatilemedia includes, for example, optical or magnetic disks, such as diskdrive 2610. Volatile media includes dynamic memory, such as systemmemory 2606. Transmission media includes coaxial cables, copper wire,and fiber optics, including wires that comprise bus 2602. Transmissionmedia may also take the form of acoustic or light waves, such as thosegenerated during radio wave and infrared data communications.

Common forms of computer readable media includes, for example, floppydisk, flexible disk, hard disk, magnetic tape, any other magneticmedium, CD-ROM, any other optical medium, punch cards, paper tape, anyother physical medium with patterns of holes, RAM, PROM, EPROM,FLASH-EPROM, any other memory chip or cartridge, carrier wave, or anyother medium from which a computer may read.

In some embodiments, execution of the sequences of instructions topractice the system is performed by a single computer system 2600.According to some embodiments, two or more computer systems 2600 coupledby communication link 2620 (e.g., LAN, PSTN, or wireless network) mayperform the sequence of instructions to practice the system incoordination with one another. Computer system 2600 may transmit andreceive messages, data, and instructions, including program, i.e.,application code, through communication link 2620 and communicationinterface 2612. Received program code may be executed by processor 2604as it is received, and/or stored in disk drive 2610, or othernonvolatile storage for later execution.

This description of the invention has been presented for the purposes ofillustration and description. It is not intended to be exhaustive or tolimit the invention to the precise form described, and manymodifications and variations are possible in light of the teachingabove. The embodiments were chosen and described in order to bestexplain the principles of the invention and its practical applications.This description will enable others skilled in the art to best utilizeand practice the invention in various embodiments and with variousmodifications as are suited to a particular use. The scope of theinvention is defined by the following claims.

1. A graphical user interface displayed on a portable wirelesselectronic device for providing context-based information services byinterpreting visual imagery on the portable wireless electronic devicecomprising: a display displaying a first tab comprising a first tab panecomprising a video display region, a first plurality of video controlsbelow the video display region, and a first text box having a first areasize, wherein the first text box overlays the video display region, thevideo display region comprises a video, and the first text box comprisesat least a portion of an augmentation information associated with thevideo, wherein the augmentation information is generated by: a userselecting text displayed in the video, recognizing the selected text inthe video, generating a context based on the selected text, and usingthe generated context in a search for relevant augmentation informationto present in the first text box; and a second tab comprising a secondtab pane comprising a second text box having a second area size, greaterthan the first area size, wherein the second text box comprises theaugmentation information displayed in the first text box, and when thesecond tab is selected, the second tab pane overlays the first tab pane,wherein the augmentation displayed in the first text box is read-onlyand the augmentation displayed in the second text box can be edited bythe user.
 2. The graphical user interface of claim 1 wherein the videodisplay region comprises a plurality of horizontal reference linesoverlaying the video.
 3. The graphical user interface of claim 1 whereinthe video comprises visual imagery captured live.
 4. The graphical userinterface of claim 1 wherein the video comprises a prerecorded visualimagery.
 5. The graphical user interface of claim 1 wherein the firstplurality of video controls comprises a play button, a stop button, azoom button, and a record button.
 6. The graphical user interface ofclaim 1 wherein the first tab comprises a first tab label, the secondtab comprises a second tab label, and the first and second tab labelsare nearer to a top edge of the display of the portable wirelesselectronic device than a bottom edge of the display.
 7. The graphicaluser interface of claim 1 wherein the portable wireless electronicdevice is a camera phone.
 8. The graphical user interface of claim 1wherein the augmentation information comprises links to on-lineinformation.
 9. The graphical user interface of claim 1 wherein thefirst text box is an editable text box.
 10. The graphical user interfaceof claim 1 wherein when the first tab is selected, the first tab paneoverlays the second tab pane.
 11. The graphical user interface of claim1 wherein the annotation is vocalized using an audio player of theportable wireless electronic device.
 12. The graphical user interface ofclaim 1 wherein the first text box is highlighted by a dashed linesurrounding the first text box.
 13. The graphical user interface ofclaim 1 wherein the first tab further comprises a second plurality ofvideo controls comprising a marking control, the marking control beingarranged to mark or highlight portions of the visual imagery.
 14. Thegraphical user interface of claim 1 wherein the augmentation informationis automatically generated without additional user input.
 15. Thegraphical user interface of claim 1 wherein the augmentation informationis generated upon explicit request by a user.
 16. The graphical userinterface of claim 1 further comprising an interface, the interfacebeing arranged to record audio information through a microphoneintegrated into the wireless electronic device.